Method of translating printed text using handheld display device

ABSTRACT

A method of translating printed text using a handheld display device having a touch-sensitive screen and a transceiver for sending and receiving digital information. The method comprises the steps of: imaging an area of the substrate containing printed text in a first language and generating image data using an optical sensor; determining interaction data using the image data, the interaction data identifying a substrate identity, the substrate identity being indexed with a page description corresponding to the printed substrate; retrieving display data using the interaction data and the page description, the display data identifying the printed text translated into a second language; and displaying, on the touch-sensitive screen, display information based on the display data. The display information comprises displayed text corresponding to the printed text translated into the second language.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a Continuation of U.S. application Ser. No.12/341,861 filed Dec. 22, 2008, which is a Continuation of U.S.application Ser. No. 10/917,436 filed Aug. 13, 2004, now issued U.S.Pat. No. 7,609,410, which is a Continuation of U.S. application Ser. No.09/722,147 filed Nov. 25, 2000, now issued as U.S. Pat. No. 6,946,672.

FIELD OF INVENTION

The present invention relates generally to devices for interacting withcomputer systems, and in particular, to a device which may be used torender otherwise passive surfaces audiovisually interactive viainvisible surface codings.

CO-PENDING APPLICATIONS

Various methods, systems and apparatus relating to the present inventionare disclosed in the following co-pending applications/granted patentsfiled by the applicant or assignee of the present inventionsimultaneously with the present invention.

6,530,339 6,631,897 7,295,839 7,593,899 7,175,079 7,064,851 6,826,5476,741,871 6,927,871 6,980,306 6,965,439 6,788,982 7,263,270 6,788,2936,946,672 7,091,960 6,792,165 7,105,753 7,182,247

The disclosures of these co-pending applications are incorporated hereinby cross-reference.

Various methods, systems and apparatus relating to the present inventionare disclosed in the following co-pending applications/granted patentsfiled by the applicant or assignee of the present invention on Oct. 20,2000:

7,190,474 7,110,126 6,813,558 6,965,454 6,847,883 7,131,058 7,533,0316,982,798 6,474,888 6,627,870 6,724,374 7,369,265 6,454,482 6,808,3306,527,365 6,474,773 6,550,997The disclosures of these co-pending applications are incorporated hereinby reference.

Various methods, systems and apparatus relating to the present inventionare disclosed in the following co-pending applications/granted patentsfiled by the applicant or assignee of the present invention on Sep. 15,2000:

6,679,420 6,963,845 6,995,859 6,720,985The disclosures of these co-pending applications are incorporated hereinby reference.

Various methods, systems and apparatus relating to the present inventionare disclosed in the following co-pending applications/granted patentsfiled by the applicant or assignee of the present invention on Jun. 30,2000:

6,824,044 6,678,499 6,976,220 6,976,035 6,766,942 7,286,113 6,922,7796,978,019 7,406,445 6,959,298 6,973,450 7,150,404 6,965,882 7,233,9247,007,851 6,957,921 6,457,883 6,831,682 6,977,751 6,398,332 6,394,5736,622,923The disclosures of these co-pending applications are incorporated hereinby cross-reference.

Various methods, systems and apparatus relating to the present inventionare disclosed in the following co-pending applications/granted patentsfiled by the applicant or assignee of the present invention on 23 May2000:

6,428,133 6,526,658 6,315,399 6,338,548 6,540,319 6,328,431 6,328,4256,991,320 6,383,833 6,464,332 6,390,591 7,018,016 6,328,417 09/575,1977,079,712 09/575,123 6,825,945 7,330,974 6,813,039 6,987,506 7,038,7976,980,318 6,816,274 7,102,772 7,350,236 6,681,045 6,728,000 7,173,7227,088,459 09/575,181 7,068,382 7,062,651 6,789,194 6,789,191 6,644,6426,502,614 6,622,999 6,669,385 6,549,935 6,987,573 6,727,996 6,591,8846,439,706 6,760,119 7,295,332 6,290,349 6,428,155 6,785,016 6,870,9666,822,639 6,737,591 7,055,739 7,233,320 6,830,196 6,832,717 6,957,76809/575,162 7,456,820 7,170,499 7,106,888 7,123,239 6,409,323 6,281,9126,604,810 6,318,920 6,488,422 6,795,215 7,154,638 6,859,289The disclosures of these co-pending applications are incorporated hereinby cross-reference.

BACKGROUND

Devices such as personal computers, personal digital assistants and evenmobile phones may be used to interact with audiovisual information andwith computer applications which have an audiovisual interface.

In general, these devices don't provide access to situated interactiveinformation or application interfaces, e.g. to an online fan club chatgroup associated with a concert poster encountered at a train station,or a mortgage calculator associated with a flyer received through themail. Each device must be used to seek out the desired information orapplication interface through a virtual space accessible through thedevice, or the information or application must be brought to the devicein a device-compatible physical format.

The present invention utilizes methods, systems and devices related to asystem referred to as “netpage”, described in our co-pendingapplications listed above, wherein invisible coded data is disposed onvarious surfaces to render the surfaces interactive in the manner ofgraphical user interfaces to computer systems.

SUMMARY OF INVENTION

In one broad form the invention provides a viewer with one or moresensors capable of sensing coded data. Images which include coded dataare sensed by the viewer and decoded. The decoded information istransmitted to a computer system which associates the decoded data withone or more files stored on the system, using previously storedassociation data. The file or files are transmitted to the viewer and tothe user via the viewer's display screen. The device includes a printingmechanism which allows markings to be made on a substrate. The markingsmay be based partially or wholly on user input or on data transmittedfrom the computer system to the viewer.

Accordingly, in one broad form, the invention provides a viewerincluding:

at least one sensor for sensing coded data on or in a substrate and forgenerating first data;

a transceiver for transmitting said first data or second data at leastpartially based on the first data to a computer system and for receivingat least display data associated with an identity derived from the firstdata from the computer system;

at least one display device for outputting visual information based atleast partially on said display data, and

a printer mechanism for printing on the substrate.

Printer mechanisms other than conventional ink deposition type printmechanisms may be used. Examples of other types of print mechanismsinclude the use of thermal paper, such as used in facsimile machines. Inaddition the substrate may be imprinted with electronically active inksand the print mechanism may be a device for selectively changing thestate of the electronically active inks For a better understanding ofelectronically active inks, reference is made to U.S. Pat. Nos.6,017,584, 6,124,851, 6,120,839, 6,120,588, 6,118,426 and 6,067,185, allassigned to E Ink Corporation, the contents of each of which areincluded herein by reference.

In one form the visual output corresponds to a human discernableinterface on the substrate, such as text.

In use the viewer may be moved across a substrate tiled with dataencoded tags and the output is modified as the viewer is moved acrossthe substrate so as to correspond to the interface on the substrate.

The optical output may correspond to the interface but be text in thesame or a different language as the interface. The optical output mayinclude the same, more or less information as the interface on thesubstrate. The interface may represent information, such as bank accountinformation which is not displayed on the interface in full but isdisplayed in full via the viewer.

The device preferably has controls to enable the optical output todisplay information with a size the same, less or more than thecorresponding information on the interface.

The device preferably includes a touch screen and the optical outputincludes interactive elements by which the user may modify the opticaloutput by interaction with the touch screen.

The touch screen may be used to input writing or other markings from theuser. The viewer may then print markings on the substrate based on theuser input markings. The markings printed may be substantially identicalto the user input markings or may be printed text corresponding to userinput handwriting after the user input has undergone text recognitionprocesses.

The device may also display motion pictures, in which case the physicalrelationship between the viewer and the substrate once the coded datahas been transmitted may be immaterial.

The viewer may also include memory into which a file or files aredownloaded for subsequent viewing.

When the viewer is lifted from the page the portion of the page withwhich the viewer was last in contact can be retained by the viewer andremain interactive. The viewer can do this by default, or alternativelyonly when the user ‘freezes’ or ‘snaps’ the current view before liftingthe viewer from the page, as discussed in more detail below. The viewermay include controls for rotating and panning the view after the viewerhas been lifted from the page, allowing the viewer to be used tonavigate an entire page after only a single contact with it. Thenavigation controls may be in the form of a small joystick, or a pair oforthogonal thumb wheels, or may be provided via the touch-screen.

The viewer senses the identity of the underlying page as well as its ownposition and orientation relative to the page using a netpage sensorembedded in the viewer. The viewer can contain multiple sensors (one ineach corner, for example), so that it works when only partiallyoverlapping a page or even when overlapping multiple pages. A singlesensor located in the center of the viewer is sufficient for mostpurposes, however, and is most economical.

The viewer is capable of providing an enhanced view of the underlyingpage. In the dark, for example, it can provide a lit view of the page.When magnification is needed, it can provide a zoomed view of the pagevia its zoom controls. The viewer may also be able to provide azoomed-out view of the page.

The viewer can provide an alternative (or additional) netpage data entrymechanism, i.e. the usual kinds of inputs, including drawing andhandwriting and gestures, can be captured relative to the displayed pagevia the touch-screen instead of relative to the physical page via thenetpage pen.

The viewer can provide access to dynamic content, such as audio andvideo, in the context of a physical page. Playback controls whichprovide access to interactive content are typically only shown when thepage is viewed through the viewer. They may be shown on the printed pageas well, although ideally in a form which clearly indicates that theyare inactive. Status information such as playing time may also be shownon-screen. The underlying page layout should provide space for thedisplay of interactive controls and status information, so that theprinted page and on-screen display remain compatible. Interactivecontrol is not limited to playback of streaming media such as audio andvideo. It can include arbitrary interaction with a computer application.It may, for example, include manipulation of an object in 3D.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a perspective view from above of an embodiment of theinvention;

FIG. 2 is a perspective view from below of the FIG. 1 device;

FIG. 3 is an exploded perspective view from above of the FIG. 1 device;

FIG. 4 is a side view from the rear of the device of FIG. 1;

FIG. 5 is a plan view from above of the FIG. 1 device;

FIG. 6 is a side view from the right of the FIG. 1 device;

FIG. 7 is a cross-sectional view taken along line AA of FIG. 5;

FIG. 8 is a perspective view from above of the internal components ofthe FIG. 1 device;

FIG. 9 is a perspective view from below of the internal components ofthe FIG. 1 device;

FIG. 10 is an exploded perspective view of the underside of the PCB ofthe FIG. 1 device;

FIG. 11 is a plan view showing the FIG. 1 device in use on a codedsubstrate in a first orientation;

FIG. 12 is a view from the side of the FIG. 11 arrangement;

FIG. 13 is a plan view of the FIG. 1 device in a second orientation onthe substrate of FIG. 11.

FIG. 14 is a plan view of the FIG. 1 device viewer showing a ‘lifted’view of a page;

FIG. 15 is a plan view of the FIG. 1 device showing a view of a page indarkness;

FIG. 16 is a plan view of the FIG. 1 device showing a ‘zoomed-in’ viewof a page;

FIG. 17 is a plan view of the FIG. 1 device showing a ‘zoomed-out’ viewof a page;

FIG. 18 is an example of a movie library page;

FIG. 19 is a plan view of a viewer showing a view of a playable videoclip;

FIG. 20 is a plan view of a viewer showing a snapped and lifted view ofa playable video clip;

FIG. 21 shows an example of a music library page;

FIG. 22 shows a plan view of a viewer showing a view of playable musicclip;

FIG. 23 shows a plan view of a viewer showing a snapped and lifted viewof a playing audio clip;

FIG. 24 shows an example of PIN advice letter from bank;

FIG. 25 shows a viewer showing a secure view of personal data;

FIG. 26 shows an example of “fill-in” form;

FIG. 27 shows a viewer allowing form fill-in by a user;

FIG. 28 shows the “fill-in” form of FIGS. 26 & 27 after printing on bythe viewer;

FIG. 29 shows a schematic of part of the viewer's electronic components,relating to file download and display;

FIG. 30 shows a schematic of part of the viewer's electronic componentsrelating to control of the printer functions;

FIG. 31 shows a document element view; and

FIG. 32 shows the viewer interaction with a netpage network.

DETAILED DESCRIPTION OF PREFERRED AND OTHER EMBODIMENTS

Referring to FIGS. 1 to 10, there is shown a viewer 100. The viewer hasa color LCD screen 102, control buttons 186, 192 & 194, a speaker 106,volume control 108, an audio out jack 110, a printer assembly and ainfrared sensing device 112. The LCD screen 102 has a touch sensitiveoverlay 132.

The viewer 100 is a netpage system enabled device and communicates witha netpage system in a similar manner to that disclosed in the co-pendingapplications referred to earlier and in particular to applications U.S.Ser. No. 09/722,142 and U.S. Ser. No. 09/721,893.

The viewer 100 has upper and lower moldings 114 and 116 respectivelywhich encapsulate a PCB 118 and all of the electronic components aremounted on or connected to this PCB. Power for the device is supplied bya rechargeable 3 volt lithium ion battery 120. The battery is not userreplaceable and is located within the casing.

The PCB 118 and attached electronics is attached to the upper molding114 by screws 125. The printer assembly is attached to the lower molding116 by screws 127, which pass through the chassis 223 into bosses 242 inthe lower molding 116.

The two moldings 114 and 116 are joined together by four screws 124,which extend through the lower molding 116 to engage the upper molding114.

The PCB includes four electrical switches which are acted on by the pushbuttons. The zoom buttons 186 allows the view to be zoomed in, zoomedout, and, when activated simultaneously, to be reset to normal scale.The snap button allows the nearest dynamic or interactive object to besnapped to the screen. It more generally allows the current view to bede-synchronized (or ‘frozen’ or ‘lifted’) from the underlying page.Optional pan controls allow the view to be panned independently of theunderlying page. The power button allows the viewer to be switched onand off. The volume control, in the form of a potentiometer, allows thevolume of the viewer's audio output to be controlled. The audio outputis used to play audio content associated with a page, and optionally toprovide operating feedback to the user.

A combined data and power connector 134 is connected to the PCB and isaccessible via opening 136 in the casing. The connector 134 includes apower input socket 138 for recharging of the battery 120 and a datasocket 140 for input/output of data to and from the device. Theconnector 134 is connected to the PCB via a flexible PCB 142 andconnectors 144 and 146.

The speaker 106 is also connected to the PCB at its lower edge viaconnector 148 and is positioned between the PCB and the lower edge ofthe casing. The speaker may be hard wired to the PCB rather than beingprovided with a removable connector.

Digital audio generated by the viewer is converted to analog via adigital-to-analog converter (DAC) 173, is amplified by an amplifier 177subject to the volume control 108, and is output to a speaker 106 or toan external audio device via an audio out jack 110. The speaker isdisabled when the audio jack is in use.

The volume control 108 and audio out jack 110 are mounted directly onthe left hand side of the PCB and are accessible via openings 154 and156 respectively in the left hand side of the casing.

The battery 120 is also positioned between the PCB and the lower edge ofthe casing. A removable connector 150 is provided to connect the batteryto the PCB. However, since the battery is not intended to be userreplaceable, a permanent connection may be used. The battery ispreferably shaped to fit the casing and accordingly has a beveled corner152.

Mounted on the lower surface of the PCB is the optical sensor device 112capable of detecting infrared markings on a substrate. The sensingdevice 112 comprises an infrared LED 160 and an image sensor 162. In useinfrared light is emitted from the LED 160 and passed through an opticalguide 164 and then through an aperture 166 in the lower molding.Reflected light passes through the aperture 166, the optical molding 164and is focused onto the CCD 162. The optical molding preferably includesa lens 168 and may include a beam splitter/combiner to allow light fromand to the LED and the CCD to follow the same path. The LED 160 may bestrobed in synchrony with image capture to prevent motion-blurring ofcaptured tag images. The image sensor 162 typically consists of a200×200 pixel CCD or CMOS image sensor with a near-infrared bandpassfilter.

The PCB also includes a processor chip 170, DRAM 172, flash ROM 174, adisplay controller 270 for controlling the LCD, a transceiver chip 178and an aerial 180.

In order to display video, the color display 102 usefully has a 4:3aspect ratio, although with the advent of DVD and digital broadcasting,digital video content with wide screen aspect ratios of 16:9 and widerare becoming more prevalent. The aspect ratio used is not critical. Theminimum useful resolution is preferably SIF resolution, which has asquare-pixel equivalent of 320×240 pixels. In order to display text, thedisplay preferably has a pitch of at least 100 pixels per inch, giving amaximum diagonal display size of four inches. Larger displays withcorrespondingly more pixels provide obvious benefits, although with theadded expense of a larger form factor and greater processing andcommunications bandwidth requirements.

The display ideally mimics the optical properties of the underlyingpaper, i.e. the paper's high reflectivity, high contrast, and Lambertianreflectance function. Although the display may be a conventionalback-lit color LCD, such as an active-matrix twisted-nematic (TN) colorLCD, it is more usefully a reflective display, such as a passive-matrixcholesteric color LCD from Kent Displays Inc, an electrophoretic displayfrom E Ink Corporation, or a rotating ball display from XeroxCorporation, Inc. It may also be a low cost organic LED (OLED) display.

Display output generated by the viewer is routed to the display 102 viathe display controller 270. Assuming a 320×240 RGB pixel display, thedisplay controller 270 has an associated or embedded 0.25 Mbytesingle-buffered or 0.5 Mbyte double-buffered display memory 181.

A dedicated compressed video and audio decoder 171 which producessquare-pixel progressive-scan digital video and digital audio output isalso provided. To handle MPEG-1 encoded video and audio, a video andaudio decoder similar to a C-Cube CL680 decoder may be used. To handleMPEG-2 encoded video and audio, a video and audio decoder similar toC-Cube's ZiVA-3 decoder may be used. An MPEG-1 decoder typically uses a4 Mbit DRAM during decoding, while an MPEG-2 decoder typically uses a 16Mbit SRAM during decoding. The decoder memory 179 may be dedicated tothe decoder, or may be part of a memory 172 shared with the processor.

The processor 175 controls and coordinates the various electroniccomponents of the viewer. The processor executes software whichmonitors, via the sensor(s) 112, the identity of the underlying page andthe position of the viewer relative to the page; communicates theidentity and position data to a netpage base station via the wirelesstransceiver 183,178; receives identity- and position-related page datafrom the base station via the transceiver; renders user output to thecolor display 102 and audio output; and interprets user input capturedvia the user interface buttons 104 and the screen's touch sensor 132.The embedded software executed by the processor is stored in thenon-volatile memory 174, which is typically a ROM and/or flash memory.Identity information unique to the viewer, as well as communicationsencryption keys, are also stored in non-volatile memory. Duringexecution the processor utilizes faster volatile memory, typically inthe form of a 256 Mbit (32 Mbyte) dynamic RAM (DRAM) 172.

The processor 175 communicates with the other components via a sharedbus 280. The processor 175, the bus 280, and any number of othercomponents may be integrated into a single chip. As indicated in FIG. 28the integrated components may include the digital transceiver controller183, the video decoder interface 187 and the tag image sensor interface185. If a fingerprint sensor 189 is provided a fingerprint sensorinterface 191 may be incorporated in the integrated components. In amore highly integrated chip, the integrated components may also includethe display controller 270, the image sensor 112, the compressed videoand audio decoder 171, the audio digital-to-analog converter (DAC) 173and the memory 172. The analog radio transceiver 178 is unlikely to beintegrated in the same chip, but may be integrated in the same package.A parallel interface 193 links the buttons 104, touch sensor 132 and theLED 160 to the bus 280.

The processor 175 is sufficiently powerful to render page content atinteractive rates, i.e. at least 10 Hz, and to transform videodecompressed by the video decoder and merge it with other page content.If it is sufficiently powerful, then it may also perform video and audiodecoding, obviating the need for a video and audio decoder.

The transceiver 178 is typically a short-range radio transceiver. It maysupport any of a number of wireless transmission standards, includingBluetooth/IEEE 802.15, IEEE 802.11, HomeRF/SWAP, HIPERLAN, and OpenAir.Bluetooth/IEEE 802.15, IEEE 802.11-1997, HIPERLAN, OpenAir, andHomeRF/SWAP all support transmission rates in the range of 1 to 2Mbit/s. IEEE 802.11b supports transmission rates of 5.5 Mbit/s and 11Mbit/s. HIPERLAN also supports a transmission rate of 24 Mbit/s in analternative mode. Beyond these currently-supported wireless LAN (WLAN)standards, next-generation WLAN standards promise to supporttransmission rates of 100 Mbit/s and higher.

The viewer may alternatively be connected to the a base station bycable, or may utilize a non-radio-frequency wireless transport, such asinfrared. IEEE 802.11, for example, optionally utilizes an infraredtransport. IrDA also utilizes an infrared transport.

The lower surface of the lower casing 16 is provided with four outerfeet 182 and four inner feet 184. The outer feet are located near thecorners of the device whilst the inner feet 184 are located at thecorners of a square centered on the lens 168.

All eight feet extend the same distance and enable the device to beplaced on a planar surface with a small gap between the surface and thegeneral plane of the lower molding. The feet have a rounded contactsurface and this aids in sliding the device across the surface. Theinner feet 184 ensure that if one or more of the outer feet 182 are notsupported the device does not tip relative to the surface. They also aidin keeping the surface flat near the sensing device 112.

The sensor device 112 is infrared sensitive. The CCD 162 is sensitive toinfrared light, either inherently or by use of filters and the LED 160emits infrared light, again inherently or by use of filters. The lens168 is focused on the plane of the inner and outer feet 182 and 184, asthis is where a substrate to be sensed will be located. The sensordevice is capable of detecting infrared absorptive tags, such as netpagetags. For a full description of the processes involved, reference ismade to our co-pending application U.S. Ser. No. 09/721,893 referred toearlier. The CCD 162, the LED 160 and processing functions incorporatedin the processor chip 170 are similar to those disclosed in theco-pending application.

The device is thus capable of sensing and decoding netpage tags on asubstrate. Image data captured by the CCD 162 is sent to the processorchip 170 and decoded to a region ID (or page ID) and a tag ID. Theregion ID and tag ID, together with other necessary information, istransmitted to the netpage system via the transceiver chip 183,178. Thenetpage system resolves the region ID and tag ID to correspondingdocument data which it transmits back to the device. The processorreceives the document data via the transceiver, and renders the data fordisplay on the color display 102 via the display controller 270.

The LCD screen 102 is overlaid with a touch sensitive overlay 132. Whenviewing information via the LCD screen, the screen will include controlsto enable the user to control the device by touching the screen with anon-marking stylus.

The viewer 100 also includes a printer assembly, which is located in thebody of the viewer.

The print assembly includes a printhead assembly 202 and an inkcartridge 201 connected together by a hose assembly 205. The printheadassembly 202 includes a printhead 203. The lower molding 116 has an inkcartridge compartment 208 to enable user replacement of the inkcartridge 201. The compartment 208 is accessed via cover 209. The hoseassembly 205 includes a connector 211 for connecting to the inkcartridge.

The print assembly is preferably an inkjet type printer and morepreferably a full-color inkjet. Accordingly the ink cartridge 201includes multiple inks. In the preferred embodiment the print assemblyis a full-color CMY or CMYK printer and the ink cartridge 201 includesthree or four separate ink chambers 252. Whilst any inkjet printhead maybe used, more preferably the printhead is a microelectromechanicalsystem (MEMS) type printhead (Memjet), such as that disclosed in ourco-pending application U.S. Ser. No. 09/575,141.

A Memjet printhead provides a combination of printhead width,compactness, low power consumption, and low cost which makes it uniquelysuitably for integration in the printing viewer. Although a Memjetprinthead provides 1600 dpi resolution, a printing viewer may get bywith less. If the printing viewer is sufficiently small, then an areaprinthead can be used rather than a scanning printhead.

The printhead assembly 202 extends across at least the full width of theLCD screen 102 and prints by moving upwards or downwards. The printheadassembly 202 is mounted on a continuous elastomeric belt 213 which runsaround rollers 215 and 217 located at the top and bottom of the screen.The lower roller 215 is driven by a DC stepper motor 286 via a worm gear221. The two rollers are mounted on a printer chassis 223. Mounted oneach side of the chassis 223 are transverse rods 225. The printheadassembly 202 is mounted on these rods 225 and these rods maintain theprinthead in the correct position; the belt 213 merely causes theprinthead assembly 202 to move backwards and forwards along the rods225.

The lower molding 116 is provided with a rectangular aperture 250 and ametal base plate 236 is attached to the underside of the molding 116.The base plate 236 has a rectangular aperture 252 sized so that theprinthead assembly 202 is not exposed when at either extreme of itstravel. The base plate is preferably attached to the lower molding 116by an adhesive.

The operation of the print assembly 202 is controlled by a dedicatedprint engine/controller (PEC) chip 281 located on the PCB 118. Anexample of a suitable PEC is described in our co-pending applicationU.S. Ser. No. 09/575,108. The PEC chip 281 generates bi-level dot datafor the printhead in real time and otherwise controls the operation ofthe printhead. Communication with the printhead assembly 202 is via aflexible PCB 228 which engages the PCB 118 via connector 230. The DCmotor 286 is connected to the PCB via a flexible PCB 231.

A master QA chip 282, for example as described in our co-pendingapplication U.S. Ser. No. 09/113,223, is provided which the PEC uses toauthenticate an identical QA chip 283 embedded in the replaceable colorink cartridge 201. A raster image processor (RIP) DSP 284 may be usedfor rendering print data at high speed. Depending on the desired printquality and speed and the performance of the processor, the RIP DSP maybe omitted with the processor performing rasterization. Depending on theneed to accurately control ink quality and to monitor consumption, andon the availability of alternate mechanisms for detecting ink depletion,the inclusion of the QA chips may not be needed.

Referring to FIG. 29, the processor 175 typically communicates with PEC281 via a serial bus 285. The serial bus 285 may be high-speed orlow-speed depending on the desired print quality and speed. Theprocessor 175 controls the stepper motor 286 which moves the printheadassembly 202 via the parallel interface 193.

The elastomeric belt 213 and the printhead assembly 202 extend acrossthe full width of the screen 102 and by necessity obscure the sensor 112as the printhead assembly moves between the rollers 215 and 217. Thebelt 213 is provided with two pairs of circular apertures 232 and 234.The first pair 232 align when the printhead assembly 202 is adjacent thedrive roller 215. The second pair 234 align when the printhead assembly202 is adjacent the other roller 217. When the apertures align thesensor 112 may sense coded data on a substrate below the viewer. Whenthe apertures 232 and 234 do not align the sensor 112 cannot sense anydata on the substrate.

The viewer may be designed so that the code sensor has a view of thesurface during printing. The code sensor itself can then be used todetect movement. Movement detected in this way can be used to terminateprinting. It can also be used to modify the print data in real time sothat the print data is properly registered with the underlying page atall times. Multiple code sensors may be used if located around theperiphery of the viewer.

An accelerometer may be included in the viewer to detect movement of theviewer during printing. If movement is detected, then printing can beimmediately terminated.

Usage

Referring to FIGS. 11 and 12, in use the device 100 is placed on asubstrate 200 having netpage tags tiled over its surface. The substrate200 may be paper, electronic paper such as used by E Ink Corporation, aplastic sheet or any other suitable substrate. As seen in FIG. 12 thesensor device 112 senses one or more of the tags, decodes the codedinformation and transmits this decoded information to the netpagesystem. In FIG. 11 the substrate carries human readable text 204 and sois readable without the device 100.

The device may be used with a surface which only carries netpage tagsand so appears as a blank document to a user without an authorizedviewer. As discussed in our earlier applications, each tag incorporatesdata which identifies the page it is on and its location within thepage. The netpage system is thus capable of determining the location ofthe viewer 100 and so can extract information corresponding to thatposition. When displaying static images or text the displayedinformation usually corresponds to the human discernable text or othermarkings 204 on the substrate. Additionally the tags include informationwhich enables the device and/or system to derive an orientation of thedevice relative to the page from the tags. This enables the informationdisplayed to be rotated relative to the device to match the orientationof the text 204. Thus information 206 displayed by the viewer appears tobe what is on the page in the corresponding position under the viewer.

When the viewer first comes into contact with a new page, it downloadsthe corresponding page description from the relevant page server. Itthen renders the viewer's view according to the current view transform,i.e. according to the viewer's current zoom setting and its position androtation relative to the underlying page. Whenever the view transformchanges, i.e. because the user moves the viewer or changes the viewer'szoom setting, the viewer re-renders the view according to the new viewtransform. For the purposes of the following discussion, the size of theviewer's window onto the page is also assumed to be part of the viewtransform.

Whenever the view transform changes, the viewer transmits the viewtransform to the netpage page server responsible for the underlyingpage. This allows the page server to commence streaming dynamic objectswhich have come into view and to cease streaming dynamic objects whichare no longer in view. It also allows the page server to provide theviewer with static objects, such as images, at a suitable resolution.

As the device is moved the sensor device 112 images the same ordifferent tags, which enables the device 100 and/or system to update thedevice's relative position on the page and to scroll the LCD display asthe device moves. The position of the viewer relative to the page caneasily be determined from the image of a single tag; as the viewer movesthe image of the tag changes and from this change in image the positionrelative to the tag can be determined. The system “knows” the absoluteposition of the tag on the page from its tag ID and so the absoluteposition of the viewer on the page may be derived. The viewer sensestags, and thereby its time-varying position relative to the substrate,and sufficiently frequently that movement of the viewer results in asmooth, as opposed to a jerky, scrolling of the display.

The information 206 displayed in the viewer is preferably aligned withthe text 204 on the page 200, so that it appears as if the viewer 100 istransparent. Thus the orientation of the viewer 100 does not alter theorientation of the displayed information 206, as seen in FIG. 13.

However, as seen in FIG. 14, the text 206 displayed on the LCD screen102 need not remain aligned with the physical text but may remainaligned with the viewer 100. The user may use the ‘snap’ button 192 toinstruct the viewer to display text aligned with the viewer, not thepage text 204.

The viewer 100, if provided with backlighting of the LCD 132, is capableof displaying information in the dark. FIG. 15 shows a page 200 with aviewer 100 displaying information 206 corresponding to information onthe page which is not visible due to lack of ambient light.

Whilst the information 206 shown on the LCD 102 is similar to theprinted information 204, it need not be identical. The left hand controlbutton 186 is a rocker type device. When pressed on the left, the buttonactivates switch 188 and when pushed on the right the button activatesswitch 190. Pushing down centrally or equally on the left and rightactivates both switches 188 and 190. Pressing on the left switch 186causes the display to zoom in whilst pressing on the right causes thedisplay to zoom out. Pressing centrally resets the display to the devicedefault, which may be a 1 to 1 ratio. FIG. 16 shows a viewer 100displaying “zoomed-in” text 206 whilst FIG. 17 shows “zoomed-out” text206, both being of text 204 on a page 200.

The device is not limited to simply reproducing the text printed on apage. Because the printed text is associated with one or more electronicdocuments in the netpage system, the device can provide more than justthe text to the user. Where a document has references, such as footnotesor hyperlinks, these references may be shown on screen. The user maythen view those references by touching the reference on screen. This ispicked up by the touch screen overlay 132 and the appropriateinformation displayed, either in a new window or as an overlay on theexisting text. This information may be displayed for a set period, afterwhich the display reverts to its original display. The system may alsocause additional information to be shown on the LCD screenautomatically. Where the page is mainly text the device 100 may be setto a read mode whereby as the user reads the page the text is recitedand output via speaker 106. This would be of benefit to people learningto read, such as children or those learning a foreign language. Using astylus and the touch screen the user indicates where they are readingand the corresponding text is spoken by the machine. Pronunciation of aword or phrase may be practiced by repeatedly drawing the stylus overthe relevant text.

The text displayed need not be the same language as the text on thepaper. The device may be set to display a single language, whenpossible. Thus if a netpage document exists in both English and Italian,for example, the device could be set to display the Italian text eventhough the printed document is in English. Where a document only existsin one language, the system may be configured to either display thedocument in that language or nothing except a message to the effect thatdocument is not available in the desired language.

If desired the device may be set to an “automatic recital” mode in whichthe screen automatically scrolls through the entire document, with orwithout a corresponding aural recital. An aural recital may also occurwithout a corresponding display of text. The speed of playback may becontrolled by the user. Since each tag on the page identifies thedocument the device merely needs to sense one tag on the page for thosefeatures to be activated.

Depending on the document size and device memory, an entire document maybe downloaded to the device or “chunks” of data may be downloaded asrequired or as expected to be required. A simple text document, even ofmany pages, is relatively small and may be buffered in its entiretywithout needing extra memory; the device is capable of playing video, aswill be discussed later, and the buffer needed to accommodate variationsin data reception for video display is more than sufficient toaccommodate most text documents. Where an entire document is downloaded,preferably the download is configured to initially send datacorresponding to the location of the viewer on the page.

When viewing text or static images the device may have two modes. Thefirst mode, described above, maintains a correlation between theinformation displayed on the LCD screen and the position of the deviceto the substrate. In the second mode, the device may be moved withoutaffecting the display; the scrolling and autoplay feature discussedabove being a subject of this feature. To set the device to the secondmode the user may press the ‘snap’ button 192. Once in the second modemovement of the device across the paper does not automatically affectthe information displayed. The information displayed continues tocorrespond to the location when the viewer was placed in this mode.Movement of information across the display in this mode is controlled bythe user. A document will continue to be displayed or be accessibleuntil the device is placed on a different netpage encoded substrate, atwhich point the device may normally revert to its first mode.Alternatively, the user must explicitly revert to the first mode bypressing the ‘snap’ button again.

FIGS. 18, 19 and 20 provide an example of the interaction of the viewerwith a video clip. FIG. 18 shows a sample ‘movie library’ page 210containing multiple video clip icons 212. Again the page is tiled withnetpage tags, not shown. FIG. 19 shows the view the viewer provides of avideo clip icon, which includes playback controls 214, which may beactivated via the viewer's touch-screen, and the playing time. Thescreen-displayed information 214,216 remains fixed relative to thedisplayed text 218 as the viewer moves. FIG. 20 shows the view theviewer shows once the nearest video clip is ‘snapped’ to the screen,i.e. with the video clip filling the screen, ready to be played. Asseen, all of the playback controls and playing time are displayed whenthe view is snapped. The user snaps the nearest interactive or dynamicobject to the centre of the screen by pressing the viewer's snap button.Alternatively or additionally, an interactive object may snap to thescreen automatically when activated, e.g. when played in the case of avideo clip. In the example, status information and controls aresuperimposed on the video clip to maximize the video clip's use of thescreen. The viewer 100 typically suppresses the display of the statusinformation 216 and playback controls 214 once the clip is playing,either until the clip terminates or until the user initiates furtherinteraction by touching the touch-screen 132. The playing time ispreferably displayed in units of hours, minutes and seconds, and ispreceded by a current chapter number if the clip has multiple chapters.The playback controls 214 include stop, play, pause, skip backwards andskip forwards. When the clip is stopped or paused, a play button isshown. When the clip is playing, a pause button is shown in place of theplay button. The skip controls skip to the start/end of the currentchapter, or, if already at the start/end, to the start of theprevious/next chapter. If held, rather than pressed momentarily, theskip controls scan backwards/forwards through the clip, i.e. they playthe clip at a greater than normal rate backwards/forwards.

As an alternative to on-screen controls, the viewer may providededicated playback control buttons which control whichever dynamicobject is current. It may also provide a dedicated display, such as amonochrome segment LCD, for the display of chapter (or track)information and playing time.

The viewer streams dynamic objects such as video clips from any numberof servers via the current netpage base station. The viewer is optimizedto only stream objects which are currently visible on the viewer'sscreen. When an object ceases to be displayed, it may be configured toautomatically pause, or it may continue to ‘play’ on the server, withoutconsuming communications bandwidth, so that when it is once againdisplayed its playback has progressed in time as expected by the user.

FIGS. 21, 22 and 23 provide an example of the interaction of the viewer100 with an audio clip. FIG. 21 shows a sample ‘music library’ page 220containing multiple audio clip entries 224.

The entries may be grouped under a title 222. FIG. 22 shows the view theviewer provides of a set of audio clip entries 224, which includes anindication of a ‘current’ audio clip entry with an associated playbutton 226 which may be activated via the viewer's touch-screen. Thecurrent entry is preferably simply the entry closest to the center ofthe screen and so changes as the viewer is moved, unless a clip isplaying. If no entry is sufficiently close, then no entry is current.FIG. 23 shows the view the viewer provides once the nearest audio clipis snapped to the screen or selected, i.e. with clip-related informationfilling the screen. Note that, as illustrated in the example, a snappedview can be quite different from any view directly derivable from thepage itself.

As described above, when the viewer is lifted from the page the portionof the page with which the viewer was last in contact remains displayedand interactive. It may also be advantageous for the user to be able to‘freeze’ the current view so that the viewer no longer synchronizes theview with the underlying page. When the user snaps the nearestinteractive or dynamic object to the screen the view is effectivelyfrozen. When the user presses the snap button again, the viewer onceagain synchronizes the view with the underlying page. When there is nointeractive or dynamic object to be snapped, the viewer may stillinterpret the snap as a freeze command, but may do no more than rotatethe current view so that it is oriented the right way up, i.e. so thatthe axes of the screen and of the displayed page are aligned. The snapbutton may therefore be used as a freeze button, but as an alternativelythe viewer may provide a freeze button separate from the snap button.

The viewer may provide a mode wherein the axes of the screen and of thedisplayed page are always kept aligned even though the viewer isotherwise synchronizing the view with the underlying page, i.e. theviewer synchronizes the view according to the position and identity ofthe underlying page, but not according to the rotation of the underlyingpage relative to the viewer. This mode has the advantage that thedisplayed view always looks sensible to the user operating the viewer.It has the further advantage that rotation is never allowed to degradethe quality of the displayed image of the page. If rotation is neverrespected by the viewer, then the viewer is also freed from ever havingto compute a rotated page image. As a compromise, the viewer may respectthe nearest multiple of ninety degrees to its actual rotation relativeto the page.

FIGS. 24 and 25 provide an example of the interaction of the viewer with‘secure’ data requiring authorized access. FIG. 24 shows an examplepersonal identification number (PIN) advice letter 230 from a bank. Boththe account number 231 and PIN 233 are suppressed on the printed page,i.e. printed as a series of X's. FIG. 14 shows the view the viewerprovides of the PIN advice letter. The view shows the actual accountnumber 237 and PIN 238, on the basis that the netpage system hasverified that the user of the viewer is the owner of the data. Theadvice letter is recorded on a netpage registration server as beingowned by a particular user, i.e. by the user to whom the letter wasoriginally addressed. The viewer is recorded as owned by the same user.The identity of the actual user is optionally verified with respect tothe identity of the owner of the viewer by verifying the fingerprint ofthe actual viewer against the fingerprint of the owner. The fingerprint,or some other biometric, is recorded by the registration server for theowner. The viewer may incorporate a fingerprint sensor 189 for thepurposes of fingerprint capture. Providing a biometric such as afingerprint or signature to the viewer may place the viewer in a‘secure’ mode wherein secure information is accessible for a period oftime. An indicator, such as a lit LED, may indicate this mode to theuser. Alternatively, secure access is only granted while the user'sbiometric is being sensed. In the example, the actual account number andPIN is then only shown while the user's finger is in contact with thefingerprint sensor.

FIGS. 26 and 27 provide an example of the interaction of the viewer witha fill-in form 240. The form has a number of fill-in fields 241. FIG. 26shows an example form. FIG. 27 shows the view the viewer provides of theform. The stylus 244 can be used to fill in the fields of the form viathe touch-screen. The example shows the form 240 being filled in byhand, pending handwriting recognition according to normal netpagemechanisms.

Netpage input, in the form of digital ink, is labeled to indicate how itwas captured, i.e. through a netpage pen interacting with a printednetpage, or through a stylus interacting with the touch-sensitive screenof a viewer. The digital ink is also labeled with any change in theauthorization state of the viewer, e.g. with respect to the presence ofthe user's finger on the viewer's fingerprint sensor. This allows thepage server, when interpreting netpage input in relation to thecorresponding page description, to ignore document elements not visibleto the user when the input was captured.

Digital ink generated through a viewer is otherwise suitably transformedto look like it came from a netpage pen interacting with the underlyingpage, i.e. the digital ink is labeled with the page ID of the underlyingpage, and the coordinates of position samples in the digital ink aretransformed from the coordinate system of the viewer into the coordinatesystem of the underlying page, according to the view transform currentwhen the position sample was captured. As an alternative, the viewer mayprovide an insertion point specifiable via the stylus, an associatedtext cursor, and text entry via an on-screen keyboard or viacharacter-wise character entry and recognition, in the manner of currentpersonal digital assistants (PDAs).

The printer is most useful in filling out forms such as that displayedin FIGS. 26 and 27. The users uses the stylus 244 to “write” informationonto the display, as shown in FIG. 28. The display is updated to reflectthe digital ink captured by the viewer.

Referring to FIG. 28, the print assembly is controlled to print ink 260in the fill-in fields of the form 240 to correspond to the digital inkcaptured by the touch screen. Thus at the end of the process anelectronic version and a hard copy version of the completed form exist.Printing of the ink 260 may occur as the user fills in each text box orit may occur after completion of the form. Printing after completionenables mistakes or alterations to be made before committing to paperbut may require the user to move the viewer across the page to enableprinting of all information. Printing as the user fills in the formavoids the need to move the device to print information. However, unlessan erasable marking system such as E-ink is used, all mistakes andalterations are printed to the page.

The printer assembly may be used to physically transfers onto theunderlying page any virtual markings made by the user via the viewertouch-screen. Virtual strokes can be faithfully reproduced, includingcolors, textures, line widths, and even subtle brushing effects such asproduced by striated brushes and airbrushes. The virtual marking mode ofthe stylus on the screen can be completely controlled in aposition-dependent manner by the active application(s), i.e. as oneexpects in any graphical user interface.

In the limit case, a user interacts with Netpages exclusively via theNetpage printing viewer, eliminating the need for a Netpage Pen.

The printing viewer is designed to print when stationary with respect tothe underlying page. Because the printhead 203 is screen-width, it cantransfer the entire contents of the screen 102 onto the underlying pagein a single pass. When not printing, the printhead assembly 202 isparked in a recess to prevent accidental damage. To conserve power, theprinthead assembly 202 can be parked at the top or bottom of the screen102, and the printhead 203 can print when moving in either direction. Toprevent user access to the cavity in which the printhead assembly moves,the assembly is integrated into a screen-width belt which also acts asthe carriage transport. The belt obscures the viewer's code sensorduring printing, but contains pairs of matching holes which line up whenthe printhead is parked, providing the code sensor with a view of thesurface. Two pairs of holes are provided, corresponding to the twoparking positions.

Handwritten text can be transferred to the underlying page before orafter it is recognized by the Netpage system and converted to computertext. If transferred after conversion, the viewer can transfer thecomputer text to the underlying page in place of the handwritten text.Similarly, a signature can be transferred to the page before or after itis used by the Netpage system to verify the identity of the user. Iftransferred after verification, the viewer can transfer a verificationicon and/or a serial number in place of the signature.

The Netpage printing viewer can also be used to paste information onto apage, including images and text. The user may take a photo with a cameraand paste it into an image field requiring the photo. The printingviewer links the photo to the image field and faithfully transfers thephoto to the page, registered with the image field. If the viewer onlypartially overlaps the field, then only the portion of the fieldaccessible to the viewer is printed with the photo. When the user movesthe viewer across the field the viewer prints additional parts of thefield. The viewer may prompt the user to move the viewer to coverun-printed portions of the field. Any un-printed portions of the imageare recorded as pending output against the page instance maintained bythe relevant Netpage page server. If a viewer comes into contact withthe image field at a future time then any un-printed portions can beprinted.

If the user pastes an image into a viewer-displayed image field when theviewer is not in contact with the page, then the entire image isrecorded as pending output.

The printing viewer may incorporate a digital camera. Once the usertakes a photo, it may “float” on the display as an icon until the userexplicitly pastes it in an image field. The viewer may provide dedicatedbuttons for browsing, viewing, editing and deleting a collection ofphotos stored in the viewer. The viewer may also support removablestorage, e.g. via a memory card socket, for storage of photos.

The user may also paste information which is known to the viewer or tothe Netpage system, such as contact details, into a Netpage form whichis not otherwise linked to the user.

The viewer may alternatively or additionally contain a mobile telephonetransceiver for longer-range communication with a netpage server via amobile telephone network. If the transceiver supports a third-generation‘always-on’ packet-switched connection, then the viewer may download orstream page content at will. If the transceiver only supports acircuit-switched connection, then the viewer may choose to connect anddownload a full page description whenever it encounters a new page,provide local navigation of the page via the downloaded pagedescription, and only make further connections when dynamic contentneeds to be displayed or when hyperlinks are activated.

If the viewer incorporates a longer-range transceiver, then it may actas a netpage base station for wireless netpage pens and other netpagesensing devices.

When used to interrogate secure data, the viewer may incorporate afingerprint sensor, such as the Vericom FPS200, to allow it tocontinuously monitor and verify the identity of the user. The processortypically interrogates the fingerprint sensor via a serial interface.

The viewer may incorporate several levels of power management. After aperiod of inactivity the viewer may inactivate the display. After alonger period of inactivity the processor may enter a power-conservingquiescent state. Power management may be coupled with the image sensorand/or the touch sensor, allowing wake-up on screen interaction ormovement. The viewer may also incorporate an accelerometer for thispurpose.

The viewer may be configured with software allowing it to function as apersonal digital assistant (PDA). The components and functions of theviewer may also be incorporated into a mobile phone.

The viewer handles a variety of page content, including styled text,outline graphics, bitmapped images, audio, and video. While audio andvideo are by their nature dynamic (i.e. time-varying), text, graphicsand images are normally static. They may, however, be part of a dynamicstream representing the output of an interactive application, executingremotely or within the viewer. A local application may be defined by ascript object which is interpreted by the viewer, e.g. coded in Java orsimilar.

Page content falls into three categories: (1) static elements whichappear on the printed page; (2) static elements which only appearthrough the viewer; and (3) dynamic elements which only appear throughthe viewer. The first category includes all the visual elements of thepage. The second category includes, for example, elements associatedwith controls for dynamic content, e.g. playback controls for a videoclip, and elements associated with secure information. The thirdcategory includes anything of a dynamic nature, e.g. a video clip. Adynamic element may have a static counterpart in the page description sothat it has a meaningful visual appearance on the printed page. A videoclip, for example, may be represented by a rectangle of the appropriatesize showing the title of the clip.

Document elements in the second and third categories have associatedview filters 245 which restrict their appearance. The ‘viewer view’ 246filter restricts the element to appearing through a viewer. The ‘secureview’ 247 filter restricts the element to appearing through anauthorized viewer. The view filter class diagram is shown in FIG. 31.

A digital ink stream is generated with a maximum data rate of about 5Kbit/s. The viewer generates a view transform stream with a similarmaximum data rate. The viewer therefore generates wirelesscommunications traffic to the base station with a maximum data rate ofabout 10 Kbit/s, which is negligible.

To minimize wireless communications traffic from the base station to theviewer, it may be mandated that only one dynamic object can be active ata time. Even though more than one object may be visible through theviewer, the page server may deactivate all but one such object. It may,for example, only leave the dynamic object closest to the center of theviewer's display active. The user may be able to explicitly activate analternative dynamic object by clicking on it with the stylus, or theuser may be required to explicitly activate any dynamic object, thusimplicitly deactivating any currently active object.

Where a dynamic object represents an application window, the page servertypically relays to the application any changes to the viewer's viewtransform which relate to the application window. This allows theapplication to generate output when the window is visible, and to tailorthe output to the actual overlap between the window and the viewer'sview. The output from the application may be in the form of a videostream, or it may be in the form of a stream of graphics commandsoperating on text, graphics and images.

It is possible to display output from a remote application through theviewer, and capture stylus input through the viewer and route it to theremote application, without the application having to be explicitlydesigned for interaction with the viewer. Applications typicallyinteract with user interface devices such as bitmapped displays,keyboards and pointing devices in a device-independent manner byutilizing the services of a windowing system, and this allows the userinterface devices to be remote from the applications without changinghow the applications interact with them.

As an alternative to rendering the view on demand, the viewer may renderthe entire page into a memory buffer and then simply display a suitablytransformed window into the buffer. However, since the rotation ofrelatively low-resolution text produces unacceptable text quality, thepage must either be rendered at a higher resolution than the displayresolution and appropriately lowpass filtered before display, or thedisplay resolution must be higher than assumed so far.

As a further alternative, the viewer may rely on the page server or thebase station to render the page and supply it to the viewer, typicallyin compressed form. In this case the viewer is still responsible forintegrating dynamic streams with the static page view.

As yet another alternative, the viewer may rely on the page server orthe base station to render the current view and supply it to the viewer,typically in compressed form. In this case the viewer need contain nosupport whatsoever for interpreting and rendering the variety ofelements which may comprise a page description. Assuming a 320×240 RGBpixel display, a 10 Hz update rate, and 10:1 compression, this requiresa data rate of about 1.8 Mbit/s, which is within the realms ofpracticality using currently-available personal-area, local-area andwide-area wireless networking technologies as discussed above.

FIG. 32 illustrates the interaction between the netpage viewer 100 andthe netpage network. The viewer's time-varying view transform, as wellas digital ink captured via the viewer, are transmitted (at A) to anetpage base station 290. The base station 290 relays (at B) the viewtransform and digital ink to the page server 291 which holds theelectronic description of the physical page with which the viewer isinteracting as identified by the page ID embedded in the tags of thepage. The page server 291 interprets the view transform and digital inkwith respect to the page, and initially responds (at C) with the pagedescription. It subsequently transmits (at C) dynamic object updates andstreaming dynamic content. The base station relays (at D) data from thepage server to the viewer.

The page server may notify (at E) an application 292 that the viewer'sview onto a window associated with the application has changed, and theapplication 292 may respond (at F) with commands instructing the viewer100 to update parts of its current view. The page server 291 relays (atC) these commands to the base station 290, which relays (at D) thecommands on to the viewer 100.

Whilst the display of the preferred embodiment is visual, a Brailleviewer may be made according to the invention. A Braille viewer isanalogous to the optical viewer. It displays in registered fashion andvia a tactile Braille display the text of a netpage. It also displaystext labels corresponding to graphics and images, where available in thepage description. Because Braille characters are typically rendered atlarger sizes than visually-recognized characters, the Braille viewertypically presents a zoomed view of the underlying page. This means thatas the viewer is moved over the page, the display moves correspondinglyfaster.

In other embodiments the viewer can be transparent when it is not incontact with a page, so that while it is being placed on a page, thepage is seen through the viewer. This can be implemented using a LCDshutter which is reflective when opaque, positioned behind atransmissive color display. However, the netpage viewer design issimplified if the viewer is always opaque, and only ever virtuallytransparent.

Whilst the invention has been described with reference to the netpagesystem which uses invisible tags, the invention is not limited to thenetpage system or the use of invisible tags. If desired, the inventionmay utilize tags or codes which are visible to the average unaided humaneye, such as bar codes. The tags need not merely encode an identitywhich is then used to look up the relevant files. The tags may encodeinstructions at a higher level. For example a tag may encode aninstruction of “play chapter 99”. If invisible tags are used they neednot be limited to the tags disclosed in relation to the netpage system.Other tagging systems are available and any suitable tagging system maybe used. The invention is not limited to the use of inks which absorbcertain wavelengths or fluoresce certain wavelengths. Magnetic inks,surface modification, including apertures, modification of the structureof the substrate itself all fall within the scope of the invention. Thesystems and methods to link the viewer of the present invention and thesource of the document data are not limited to netpage systems. A viewermay be linked by a cable to a single computer, rather than a network ofcomputers.

The present invention has been described with reference to a preferredembodiment and number of specific alternative embodiments. However, itwill be appreciated by those skilled in the relevant fields that anumber of other embodiments, differing from those specificallydescribed, will also fall within the spirit and scope of the presentinvention. Accordingly, it will be understood that the invention is notintended to be limited to the specific embodiments described in thepresent specification, including documents incorporated bycross-reference as appropriate. The scope of the invention is onlylimited by the attached claims.

1. A method of translating printed text using a handheld display devicehaving an image sensor, a transceiver and an opaque touch-sensitivescreen, said method comprising the steps of: overlaying said device onpart of said printed substrate; imaging an area of a printed substratecontaining printed text in a first language and generating image datausing the image sensor; determining interaction data using said imagedata, said interaction data identifying a page description correspondingto the printed substrate; sending the interaction data to a computersystem, thereby causing the computer system to identify the pagedescription and retrieve display data corresponding to said printed texttranslated into a second language; receiving the display data from thecomputer system; displaying, on said opaque touch-sensitive screen,display information based on said display data, said display informationcomprising displayed text corresponding to said printed text translatedinto said second language, said displayed text being aligned with theprinted text underlying said screen, such that the opaquetouch-sensitive screen has real-time virtual transparency through thescreen to the printed content underlying said screen from a user'sperspective to provide a virtual window to the printed contentunderlying said screen.
 2. The method of claim 1, wherein said device isa mobile phone or a personal digital assistant.
 3. The method of claim1, wherein said transceiver is at least one of: a short-range wirelesstransceiver and a mobile phone transceiver.
 4. The method of claim 1,wherein said interaction data identifies a position of the devicerelative to the printed substrate and said retrieved display datacorresponds to a part of the substrate containing said position.
 5. Themethod of claim 1, wherein said device comprises a memory and saidmethod further comprises the step of: storing the retrieved display datain said memory.
 6. The method of claim 1 further comprising the step of:interacting with said touch-sensitive screen so as to zoom in on saiddisplayed information; and interacting with said touch-sensitive screenso as to zoom out from said zoomed-in information.
 7. The method ofclaim 1, further comprising the step of: interacting with the displayinformation via said touch-sensitive screen using a finger or a stylusassociated with said device.
 8. The method of claim 1 further comprisingthe step of: detecting movement of the device using an accelerometer. 9.The method of claim 1, further comprising the step of: snapping saiddisplay information, wherein snapped display information is displayedirrespective of a position of said device relative to said substrate.10. The method of claim 9, wherein snapped display information isinteractive via said touch-sensitive display screen irrespective of theposition of said device relative to said substrate.
 11. The method ofclaim 9 further comprising the step of: navigating a display page viasaid snapped display information, said display page corresponding to theprinted text of the substrate translated into said second language. 12.The method of claim 11, wherein said navigating includes panning acrosssaid display page.
 13. The method of claim 12, wherein said panning isby means of a panning interaction with said touch-sensitive screen. 14.The method of claim 1, wherein said device comprises a power managementsystem for powering down said display screen during periods ofinactivity, said power management system being coupled with at least oneinput sensor for initiating powering up of said device, said inputsensor being selected from the group consisting of: said touch-sensitivedisplay screen, said optical sensor and an accelerometer.
 15. The methodof claim 1 further comprising the step of: printing onto said substrateusing a printer integrated with said handheld display device.
 16. Themethod of claim 1, wherein said printed substrate comprises coded datadisposed thereon or therein, said coded data identifying a substrateidentity indexed with said page description.
 17. The method of claim 16,wherein said coded data further identifies a plurality of locations onthe substrate.